Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve][broker, build, client, fn, io, ml, proxy, sql, test] Merge apache/branch-3.1 into 3.1_ds #212

Merged
merged 84 commits into from
Dec 20, 2023

Conversation

nikhil-ctds
Copy link
Collaborator

@nikhil-ctds nikhil-ctds commented Dec 14, 2023

Motivation

Explain here the context, and why you're making that change. What is the problem you're trying to solve.
This PR merges apache/branch-3.1 into 3.1_ds.
3.1_ds is currently on par with apache/v3.1.1, this merge makes the branch on par with apache/branch-3.1 which is at release v3.1.2

Modifications

Describe the modifications you've done.

  • Updated GroupID's of dependencies to com.datastax.oss
  • Merged commits from branch-3.1

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (yes)
  • The public API: (no)
  • The schema: (no)
  • The default values of configurations: (yes)
  • The wire protocol: (no)
  • The rest endpoints: (no)
  • The admin cli options: (no)
  • Anything that affects deployment: (no)

Documentation

Check the box below or label this PR directly (if you have committer privilege).

Need to update docs?

  • doc-required

    (If you need help on updating docs, create a doc issue)

  • no-need-doc

    (Please explain why)

  • doc

    (If this PR contains doc changes)

lhotari and others added 24 commits December 20, 2023 21:18
…g from BookKeeper to Pulsar. (apache#21340)

There is no testing for AutoRecovery replication in Pulsar's current test suite, and we need to cover it. So migrate the replication testing from BookKeeper to Pulsar.
Fix zookeeper related flacky test

(cherry picked from commit 9ab7417)
apache#21155)

The client assumed the connection was inactive, but the Broker assumed the connection was fine. The Client tried to  use a new connection to reconnect a producer, then got an error `Producer with name 'st-0-5' is already connected to topic`.

- In a connection, the second connection waits for the first connection to complete\. But there is a bug that causes this mechanism to fail\.
- If a producer uses a default name, the second registration will override the first one. But it can not override the first one if it uses a specified producer name\. I think this mechanism is to prevent a client from creating two producers with the same name. However, method `Producer.isSuccessorTo` has checked the `producer-id`, and the `producer-id` of multiple producers created by the same client are different. So this mechanism can be deleted.

- For `issue 1`: If a producer with the same name tries to use a new connection, async checks the old connection is available. The producers related to the connection that is not available are automatically cleaned up.

- For `issue 2`:
  -  Fix the bug that causes a complete producer future will be removed from `ServerCnx`.
  - Remove the mechanism that prevents a producer with a specified name from overriding the previous producer.

(cherry picked from commit bda16b6)
(cherry picked from commit 39235ed)

# Conflicts:
#	pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ClusterMigrationTest.java
### Motivation
As https://pulsar.apache.org/docs/3.1.x/sql-overview/, Pulsar SQL is based on [Trino (formerly Presto SQL)](https://trino.io/), which supports UUID type. But now, the UUID field in Avro or JSON schema will be interpreted as VARCHAR.

### Modifications

Support decoding UUID form AVRO or JSON schema.

(cherry picked from commit 8c70943)
…elete heartbeat topic (apache#21360)

Co-authored-by: fanjianye <[email protected]>
Co-authored-by: Jiwei Guo <[email protected]>
…d avoid recursive update error (apache#21282)

(cherry picked from commit aecdb03)
liangyepianzhou and others added 27 commits December 20, 2023 21:52
…he#21447)

### Motivation

Since the files will be checked for `LICENSE` whose name is a suffix with `.conf`. But we can not add a LICENSE definition for the file `schema_example.conf` because it is a `JSON` file.

Note: the file `schema_example.conf` is only used for the integration test `CLITest.java`.

### Modifications

Rename `schema_example.conf` to `schema_example.json`.
… if the previous one is inactive (apache#21220)

If a producer establishes a new connection when it is reconnecting, while the previous connection is now inactive, the initial request made on the new connection will fail.  This failure will trigger the topic of cleaning up the inactive producers.  However, upon making a second request, the producer will be able to successfully establish a connection and proceed with the operation.

Make the initial request made on the new connection success.

(cherry picked from commit 711b621)
### Motivation

When I did this test below and got the error "Topic not found".

```java
pulsarAdmin.topics().createNonPartitionedTopic("persistent://public/default/tp_1");
Consumer consumer = null;
Consumer consumer = pulsarClient.newConsumer()
          .topic("persistent://public/default/tp_1")
          .subscriptionName("s1")
          .enableRetry(true)
          .subscribe();
```

I do create the topic `persistent://public/default/tp_1` first but got a response "Topic not found", it is confusing.

The root cause is the retry letter topic `persistent://public/default/tp_1-sub1-RETRY` was not created.

### Modifications

clear the vague response if the topic is not founded.

(cherry picked from commit 1a024bc)
…n on non-persistent topic (apache#21592)

Fixes apache#21568

Motivation
Fix incorrect unack count when using shared subscription on non-persistent topic

Modifications
In the case of a non-persistent topic, the consumer does not send an ack to the broker (see org.apache.pulsar.client.impl.NonPersistentAcknowledgmentGroupingTracker# addAcknowledgment)

To work around this, we can update unackedMessages when the broker sends a message to the consumer successfully.
Co-authored-by: Lari Hotari <[email protected]>
Co-authored-by: xiangying <[email protected]>
Co-authored-by: Jiwe Guo <[email protected]>
mvn initialize fails to discover all the sub-modules if the release versions are not available in mvn repo.
This would fail only incase of new releases where the artifacts are not yet published into the mvn repo.

Error: Failed to execute goal on project pulsar-client-all-shade-test: Could not resolve dependencies for project org.apache.pulsar.tests:pulsar-client-all-shade-test:jar:3.1.2: Failed to collect dependencies at com.datastax.oss:pulsar-client-all:jar:3.1.2: Failed to read artifact descriptor for com.datastax.oss:pulsar-client-all:jar:3.1.2: Could not transfer artifact com.datastax.oss:pulsar-client-all:pom:3.1.2 from/to
…pache#21684)

Fixes apache#21557

### Motivation

There is a network package loss issue after enabling `haProxyProtocolEnabled`, which leads the error `Checksum failed on the broker` and `Adjusted frame length exceeds`, you can reproduce the issue by the test `testSlowNetwork`.

### Modifications

Fix the bug.

(cherry picked from commit 6e18874)
@nikhil-ctds nikhil-ctds force-pushed the 3.1_ds_merge-branch-3.1 branch from 908f611 to 96d9a32 Compare December 20, 2023 16:24
@srinath-ctds srinath-ctds merged commit 7fe8c82 into 3.1_ds Dec 20, 2023
49 of 50 checks passed
@srinath-ctds srinath-ctds deleted the 3.1_ds_merge-branch-3.1 branch December 20, 2023 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.